{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lesson 3\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Heuristic Portfolio Selection "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](Lesson3GoalHeaderImage.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The universe of stocks, both local and global,  can truly baffle investors who wish to make a modest selection of stocks for their portfolios. The vastness of the choices and the  diverse behavioural characteristics of each of these stocks with respect to itself and to one another, can render a prudent selection of stocks  to be a  daunting task indeed!  \n",
    "  \n",
    "Fig. 3.1 illustrates an investor's perception of the challenging task of portfolio selection. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](Lesson3Fig3_1.png)\n",
    "#### Fig. 3.1 The challenging task of Portfolio Selection - an investor's perception "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, there is this adage, \"**never put all your eggs in the same basket**\"! So, how would an investor know that the stocks picked do not perform similarly, warranting a risk when all stocks react similarly to market events thereby impacting the performance of the portfolio? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several answers available to these questions independently and collectively from traditional investment finance, but a commonly agreed upon answer is **Diversification**. \n",
    "Diversification involves investments in different assets or asset classes or markets. A portfolio that comprises such a diversified set of securities can go a long way in mitigating risk since the securities would react differently to market events. \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The choice of assets in an investor's portfolio reflects the investor's *risk appetite* or the willingness to take risk.  It is an established investment principle that investments giving high returns are always associated with a probability of high risks and those that are less risky are associated with a probability of low returns.     \n",
    "  \n",
    "Therefore, while *risk averse* investors, as a matter of fact,  would choose to invest a larger share of their capital in risk-free instruments - government bonds, for example,  which are notionally risk-free, *risk-seeking* investors on the other hand, would choose to invest a lion's share of their capital in risky instruments -  equities or currencies or commodities, for example,  which are prone to yield quick and high returns and therefore are inherently risky.  \n",
    "  \n",
    "Investors therefore,  may hold  portfolios constituted by a mix of securities that can render the portfolio investments to  range from *less risky* through *moderately risky* to *highly risky*, as measured by a \"riskometer\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fig. 3.2 graphically illustrates some investors' portfolio \"doughnuts\" with the investors' risk appetites and approach to investments symbolically described by the icons superscribed over the investor labels.  Thus,  Investor C seems to be *risk averse* considering the larger investments made in bonds, which are risk-free, when compared to equities which are generally risky. Investor B turns out to be the opposite owning a *high risk* portfolio,  that invests a lion's share of the capital in equities. Investor D and Investor A, both owning chequered portfolios that invest lesser in bonds and more on other securities,   could be deemed to be holding *moderately high* and  *high risk* portfolios, respectively. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](Lesson3Fig3_2.png)\n",
    "#### Fig. 3.2 Portfolio selection vs Investors' risk appetites "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Diversification Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **Diversification Index** quantifies diversification. There are several diversification indices discussed in the literature. **Diversification Ratio** proposed and patented by Yves Choueifaty in 2008 [CHO 08, CHO 13], is a diversification index of recent origin,  built on the inter-dependence between assets of a portfolio. Diversification Ratio is the *ratio of the weighted sum of individual asset volatilities to the portfolio's volatility*. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let N be the number of assets in the  portfolio spanning different  asset classes or belonging to a specific class. Let $(\\bar{w}=(w_1,w_2,...w_N) )$   be the weights or the proportion of capital to be invested in individual assets in the portfolio and $\\bar{w}'$ its transpose. Let $(\\bar{\\sigma}=(\\sigma_1,\\sigma_2,...\\sigma_N))$ be the standard deviations of returns on the assets and  _V_,  the variance-covariance matrix of returns on the assets. The Diversification Ratio of a portfolio is given as follows:  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](Lesson3Eqn3_1.png)\n",
    "<h5 align=\"right\">..........(3.1)</h5>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A portfolio that is *most diversified* would yield the *maximal Diversification Ratio*. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3  Clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " A one-shot solution to ensure prudent selection of assets from a stock universe,  which will ensure benefits of Diversification is **Clustering** or **Cluster Analysis**. Clustering deals with the task of grouping a set of physical or abstract objects into classes such that objects within a class exhibit close **similarity** to one another, while simultaneously expressing a strong **dissimilarity** to objects with other classes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fig. 3.3 graphically illustrates clustering of objects represented by points,  into three clusters. It can be easily seen that those points that are in proximity to one another have  grouped themselves into a cluster. Marking the points in each of these clusters with different symbols, visibly shows within-class similarity of objects in the cluster and out-of-class dissimilarity to objects in other clusters.      \n",
    "  \n",
    "Thus, clustering exploits the characteristic of objects with similar features or behaviour,  gravitating towards one another, while moving away or repelling the influence of  objects which are dissimilar to them in features or behaviour, thereby satisfying the  adage, \"*birds of a feather flock together*\".  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](Lesson3Fig3_3.png)\n",
    "#### Fig. 3.3 Graphical illustration of clustering objects represented by points,  into groups "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Several clustering methods have been explored. **$k$-means clustering** is a cluster analysis technique which groups $N$ objects into ***k*** clusters, where $k$ is an input decided by the user. In the case of portfolio selection, if $N$ were to indicate the size of the stock universe, $k$ could indicate the portfolio size and the investor can use the input $k$ to his or her advantage to make a choice of small portfolios (typically $k\\le30$) or a large portfolio (typically $k >30$)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 Case Study"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us restrict our discussion to selection of equity stocks, after all any asset allocation plan involves significant portion of the capital being invested in equities. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We shall consider a \"mini\" stock universe of 29 equity stocks of Dow Jones Industrial Average (DJIA) Index viz., Apple (AAPL), American Express (AXP), Boeing (BA), Caterpillar (CAT), Cisco Systems (CSCO), \tChevron (CVX), \tWalt Disney (DIS), Goldman Sachs (GS), \tThe Home Depot (HD),  \tIBM (IBM), \tIntel (INTC), Johnson & Johnson\t(JNJ), \tJP Morgan Chase (JPM), Coca-Cola (KO), \tMcDonald's (MCD), 3M(MMM), Merck & Co\t(MRK), Microsoft (MSFT), Nike\t(NKE), \tPfizer (PFE), Procter & Gamble\t(PG), \tTravelers (TRV), United Health Group\t(UNH), United Technologies\t(UTX), Verizon\t(V), Verizon (VZ), Walgreens Boots Alliance (WBA), Walmart (WMT), \tExxon Mobil (XOM).   \n",
    "\n",
    "  \n",
    "The data set considered is from April 11, 2014 to April 11, 2019. Fig. 3.4 illustrates a snapshot of the DJIA dataset."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](Lesson3Fig3_4.png)\n",
    "#### Fig. 3.4 Snapshot of the DJIA dataset "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An investor desires to invest in 15 stocks from this \"mini\" universe. The following questions arise:   \n",
    "\n",
    " > How can the investor decide on which combination of assets among the 29 stocks, is the best?   \n",
    " > How can diversification be ensured,  when the assets belong to different sectors and therefore behave differently under varying market conditions?  \n",
    "  \n",
    " $k$-means clustering provides solutions to both questions at once. The following steps and the companion Python code illustrate this **heuristic portfolio selection process**. The Python code employs NumPy, Pandas and scikit-learn to effect portfolio selection using $k$-means clustering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 1:** ***Undertake data wrangling of the original stock dataset to keep it fit for further processing***.   \n",
    "\n",
    "(Refer *Lesson2 Some glimpses of financial data wrangling* to learn about aspects of financial data wrangling).  \n",
    "We assume that the DJIA dataset for the \"mini\" universe of 29 stocks  is already cleaned and the Python code shown below reads the CSV file concerned. **clusters = 15** represents the portfolio size desired by the investor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['AAPL', 'AXP', 'BA', 'CAT', 'CSCO', 'CVX', 'DIS', 'GS', 'HD', 'IBM', 'INTC', 'JNJ', 'JPM', 'KO', 'MCD', 'MMM', 'MRK', 'MSFT', 'NKE', 'PFE', 'PG', 'TRV', 'UNH', 'UTX', 'V', 'VZ', 'WBA', 'WMT', 'XOM']\n",
      "1259 29\n",
      "[[ 74.23     84.54    122.07    ...  64.26     76.5      96.72   ]\n",
      " [ 74.52571  85.5     123.25    ...  65.67     77.38     97.86   ]\n",
      " [ 73.99429  86.04    124.27    ...  66.01     76.88     98.68   ]\n",
      " ...\n",
      " [199.5     109.85    369.04001 ...  54.5      98.69     81.93   ]\n",
      " [200.61999 110.16    364.94    ...  54.51     99.6      81.56   ]\n",
      " [198.95    109.85    370.16    ...  53.44    100.8      81.95   ]]\n"
     ]
    }
   ],
   "source": [
    "#read stock prices from a cleaned DJIA dataset \n",
    "\n",
    "#Dependencies \n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.cluster import KMeans \n",
    "\n",
    "#input stock prices data set \n",
    "stockFileName = 'DJIA_Apr112014_Apr112019.csv'\n",
    "originalRows = 1259   #excluding header\n",
    "originalColumns = 29  #excluding date\n",
    "clusters = 15\n",
    "\n",
    "#read stock dataset into a dataframe \n",
    "df = pd.read_csv(stockFileName,  nrows= originalRows)\n",
    "\n",
    "#extract asset labels\n",
    "assetLabels = df.columns[1:originalColumns+1].tolist()\n",
    "print(assetLabels)\n",
    "\n",
    "#extract stock prices excluding header and trading dates\n",
    "dfStockPrices = df.iloc[0:, 1:]\n",
    "\n",
    "#store stock prices as an array\n",
    "arStockPrices = np.asarray(dfStockPrices)\n",
    "[rows, cols]= arStockPrices.shape\n",
    "print(rows, cols)\n",
    "print(arStockPrices)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 2:** ***Compute asset returns of the stocks***  \n",
    "\n",
    "(Refer *Lesson 1 Fundamentals of Risk and Return of a Portfolio* to know about stock returns computing)  \n",
    "Function **StockReturnsComputing** computes the daily return of stocks in the DJIA index and stores it in the array **arReturns**, as shown in the Python code fragment given below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#function for Stock Returns computing \n",
    "def StockReturnsComputing(StockPrice, Rows, Columns):\n",
    "    \n",
    "    import numpy as np\n",
    "    \n",
    "    StockReturn = np.zeros([Rows-1, Columns])\n",
    "    for j in range(Columns):  # j: Assets\n",
    "        for i in range(Rows-1):     #i: Daily Prices\n",
    "            StockReturn[i,j]=((StockPrice[i+1, j]-StockPrice[i,j])/StockPrice[i,j])\n",
    "\n",
    "    return StockReturn\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of the array of daily returns of stocks:\n",
      " (1258, 29)\n",
      "Array of daily returns of stocks\n",
      " [[ 0.00398  0.01136  0.00967 ...  0.02194  0.0115   0.01179]\n",
      " [-0.00713  0.00632  0.00828 ...  0.00518 -0.00646  0.00838]\n",
      " [ 0.00203  0.01581  0.01424 ...  0.00227  0.00442  0.01277]\n",
      " ...\n",
      " [-0.003   -0.00768 -0.01463 ... -0.01017 -0.00544 -0.01289]\n",
      " [ 0.00561  0.00282 -0.01111 ...  0.00018  0.00922 -0.00452]\n",
      " [-0.00832 -0.00281  0.0143  ... -0.01963  0.01205  0.00478]]\n"
     ]
    }
   ],
   "source": [
    "#compute daily returns of all stocks in the mini universe\n",
    "arReturns = StockReturnsComputing(arStockPrices, rows, cols)\n",
    "print('Size of the array of daily returns of stocks:\\n', arReturns.shape)\n",
    "print('Array of daily returns of stocks\\n',  arReturns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 3:** ***Compute mean returns and variance-covariance matrix of returns***  \n",
    "\n",
    "(Refer *Lesson 1 Fundamentals of Risk and Return of a Portfolio* to know about computing the mean returns and variance-covariance matrix of stock returns)  \n",
    "**meanReturns** and **covReturns** store the outputs of the respective computations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean returns:\n",
      " [ 0.0009   0.00029  0.001    0.00039  0.00081  0.00016  0.0004   0.00033\n",
      "  0.00085 -0.00016  0.00073  0.00032  0.0006   0.00019  0.00057  0.00044\n",
      "  0.00036  0.001    0.0008   0.00034  0.00025  0.00043  0.00095  0.00019\n",
      "  0.00101  0.00023 -0.00002  0.00029 -0.00006]\n",
      "Size of Variance-Covariance matrix of returns:\n",
      " (29, 29)\n",
      "Variance-Covariance matrix of returns:\n",
      " [[0.00024 0.00007 0.0001  0.0001  0.0001  0.00007 0.00007 0.0001  0.00007\n",
      "  0.00007 0.00011 0.00005 0.00008 0.00003 0.00005 0.00007 0.00005 0.00012\n",
      "  0.00008 0.00005 0.00004 0.00005 0.00008 0.00007 0.0001  0.00003 0.00007\n",
      "  0.00004 0.00006]\n",
      " [0.00007 0.00016 0.00008 0.0001  0.00007 0.00006 0.00006 0.00011 0.00007\n",
      "  0.00007 0.00007 0.00005 0.0001  0.00003 0.00004 0.00007 0.00006 0.00008\n",
      "  0.00007 0.00005 0.00003 0.00005 0.00007 0.00007 0.00008 0.00003 0.00007\n",
      "  0.00004 0.00005]\n",
      " [0.0001  0.00008 0.00023 0.00013 0.00009 0.00008 0.00007 0.00011 0.00007\n",
      "  0.00008 0.0001  0.00006 0.0001  0.00004 0.00005 0.00009 0.00006 0.00009\n",
      "  0.00008 0.00005 0.00004 0.00006 0.00007 0.00009 0.00009 0.00004 0.00007\n",
      "  0.00005 0.00007]\n",
      " [0.0001  0.0001  0.00013 0.00027 0.0001  0.00012 0.00007 0.00013 0.00008\n",
      "  0.00009 0.00012 0.00005 0.00012 0.00004 0.00005 0.00011 0.00006 0.00011\n",
      "  0.00008 0.00006 0.00004 0.00007 0.00007 0.0001  0.0001  0.00004 0.00007\n",
      "  0.00004 0.00011]\n",
      " [0.0001  0.00007 0.00009 0.0001  0.00018 0.00008 0.00007 0.00009 0.00007\n",
      "  0.00008 0.00012 0.00006 0.00009 0.00004 0.00005 0.00008 0.00006 0.00011\n",
      "  0.00008 0.00006 0.00004 0.00006 0.00007 0.00007 0.00009 0.00005 0.00007\n",
      "  0.00006 0.00007]\n",
      " [0.00007 0.00006 0.00008 0.00012 0.00008 0.00019 0.00006 0.00009 0.00006\n",
      "  0.00007 0.00008 0.00005 0.00009 0.00004 0.00004 0.00007 0.00006 0.00008\n",
      "  0.00005 0.00005 0.00004 0.00006 0.00006 0.00006 0.00007 0.00005 0.00006\n",
      "  0.00004 0.00013]\n",
      " [0.00007 0.00006 0.00007 0.00007 0.00007 0.00006 0.00014 0.00008 0.00006\n",
      "  0.00006 0.00007 0.00004 0.00007 0.00003 0.00004 0.00006 0.00005 0.00007\n",
      "  0.00007 0.00005 0.00004 0.00005 0.00006 0.00005 0.00007 0.00004 0.00006\n",
      "  0.00004 0.00006]\n",
      " [0.0001  0.00011 0.00011 0.00013 0.00009 0.00009 0.00008 0.00021 0.00008\n",
      "  0.00008 0.0001  0.00005 0.00016 0.00003 0.00005 0.00008 0.00007 0.0001\n",
      "  0.00008 0.00006 0.00004 0.00008 0.00008 0.00008 0.0001  0.00004 0.00008\n",
      "  0.00004 0.00008]\n",
      " [0.00007 0.00007 0.00007 0.00008 0.00007 0.00006 0.00006 0.00008 0.00014\n",
      "  0.00006 0.00007 0.00005 0.00008 0.00003 0.00005 0.00006 0.00005 0.00008\n",
      "  0.00008 0.00005 0.00004 0.00005 0.00007 0.00006 0.00008 0.00004 0.00007\n",
      "  0.00006 0.00005]\n",
      " [0.00007 0.00007 0.00008 0.00009 0.00008 0.00007 0.00006 0.00008 0.00006\n",
      "  0.00016 0.00008 0.00005 0.00008 0.00004 0.00004 0.00007 0.00006 0.00009\n",
      "  0.00006 0.00005 0.00004 0.00005 0.00006 0.00007 0.00008 0.00004 0.00005\n",
      "  0.00004 0.00006]\n",
      " [0.00011 0.00007 0.0001  0.00012 0.00012 0.00008 0.00007 0.0001  0.00007\n",
      "  0.00008 0.00025 0.00006 0.00009 0.00004 0.00004 0.00009 0.00006 0.00013\n",
      "  0.00007 0.00006 0.00004 0.00006 0.00007 0.00008 0.00009 0.00005 0.00007\n",
      "  0.00005 0.00007]\n",
      " [0.00005 0.00005 0.00006 0.00005 0.00006 0.00005 0.00004 0.00005 0.00005\n",
      "  0.00005 0.00006 0.0001  0.00005 0.00004 0.00004 0.00006 0.00006 0.00006\n",
      "  0.00005 0.00006 0.00004 0.00005 0.00006 0.00005 0.00005 0.00004 0.00006\n",
      "  0.00004 0.00005]\n",
      " [0.00008 0.0001  0.0001  0.00012 0.00009 0.00009 0.00007 0.00016 0.00008\n",
      "  0.00008 0.00009 0.00005 0.00017 0.00003 0.00005 0.00008 0.00007 0.00009\n",
      "  0.00007 0.00006 0.00004 0.00008 0.00008 0.00008 0.00009 0.00004 0.00007\n",
      "  0.00004 0.00008]\n",
      " [0.00003 0.00003 0.00004 0.00004 0.00004 0.00004 0.00003 0.00003 0.00003\n",
      "  0.00004 0.00004 0.00004 0.00003 0.00008 0.00004 0.00004 0.00004 0.00004\n",
      "  0.00004 0.00003 0.00005 0.00004 0.00003 0.00003 0.00004 0.00004 0.00004\n",
      "  0.00003 0.00003]\n",
      " [0.00005 0.00004 0.00005 0.00005 0.00005 0.00004 0.00004 0.00005 0.00005\n",
      "  0.00004 0.00004 0.00004 0.00005 0.00004 0.00011 0.00004 0.00004 0.00006\n",
      "  0.00005 0.00003 0.00004 0.00004 0.00004 0.00004 0.00005 0.00004 0.00004\n",
      "  0.00004 0.00004]\n",
      " [0.00007 0.00007 0.00009 0.00011 0.00008 0.00007 0.00006 0.00008 0.00006\n",
      "  0.00007 0.00009 0.00006 0.00008 0.00004 0.00004 0.00012 0.00006 0.00008\n",
      "  0.00006 0.00006 0.00004 0.00006 0.00006 0.00007 0.00007 0.00004 0.00006\n",
      "  0.00004 0.00006]\n",
      " [0.00005 0.00006 0.00006 0.00006 0.00006 0.00006 0.00005 0.00007 0.00005\n",
      "  0.00006 0.00006 0.00006 0.00007 0.00004 0.00004 0.00006 0.00015 0.00006\n",
      "  0.00005 0.00008 0.00004 0.00005 0.00006 0.00005 0.00006 0.00005 0.00006\n",
      "  0.00005 0.00006]\n",
      " [0.00012 0.00008 0.00009 0.00011 0.00011 0.00008 0.00007 0.0001  0.00008\n",
      "  0.00009 0.00013 0.00006 0.00009 0.00004 0.00006 0.00008 0.00006 0.00021\n",
      "  0.00008 0.00007 0.00005 0.00006 0.00008 0.00008 0.00012 0.00005 0.00008\n",
      "  0.00005 0.00007]\n",
      " [0.00008 0.00007 0.00008 0.00008 0.00008 0.00005 0.00007 0.00008 0.00008\n",
      "  0.00006 0.00007 0.00005 0.00007 0.00004 0.00005 0.00006 0.00005 0.00008\n",
      "  0.00022 0.00005 0.00004 0.00005 0.00007 0.00006 0.00008 0.00004 0.00007\n",
      "  0.00005 0.00005]\n",
      " [0.00005 0.00005 0.00005 0.00006 0.00006 0.00005 0.00005 0.00006 0.00005\n",
      "  0.00005 0.00006 0.00006 0.00006 0.00003 0.00003 0.00006 0.00008 0.00007\n",
      "  0.00005 0.00012 0.00004 0.00005 0.00007 0.00005 0.00006 0.00004 0.00006\n",
      "  0.00004 0.00005]\n",
      " [0.00004 0.00003 0.00004 0.00004 0.00004 0.00004 0.00004 0.00004 0.00004\n",
      "  0.00004 0.00004 0.00004 0.00004 0.00005 0.00004 0.00004 0.00004 0.00005\n",
      "  0.00004 0.00004 0.00009 0.00004 0.00004 0.00004 0.00004 0.00004 0.00004\n",
      "  0.00004 0.00004]\n",
      " [0.00005 0.00005 0.00006 0.00007 0.00006 0.00006 0.00005 0.00008 0.00005\n",
      "  0.00005 0.00006 0.00005 0.00008 0.00004 0.00004 0.00006 0.00005 0.00006\n",
      "  0.00005 0.00005 0.00004 0.00011 0.00006 0.00006 0.00006 0.00004 0.00006\n",
      "  0.00004 0.00005]\n",
      " [0.00008 0.00007 0.00007 0.00007 0.00007 0.00006 0.00006 0.00008 0.00007\n",
      "  0.00006 0.00007 0.00006 0.00008 0.00003 0.00004 0.00006 0.00006 0.00008\n",
      "  0.00007 0.00007 0.00004 0.00006 0.00017 0.00006 0.00007 0.00004 0.00008\n",
      "  0.00005 0.00005]\n",
      " [0.00007 0.00007 0.00009 0.0001  0.00007 0.00006 0.00005 0.00008 0.00006\n",
      "  0.00007 0.00008 0.00005 0.00008 0.00003 0.00004 0.00007 0.00005 0.00008\n",
      "  0.00006 0.00005 0.00004 0.00006 0.00006 0.00013 0.00007 0.00004 0.00006\n",
      "  0.00004 0.00006]\n",
      " [0.0001  0.00008 0.00009 0.0001  0.00009 0.00007 0.00007 0.0001  0.00008\n",
      "  0.00008 0.00009 0.00005 0.00009 0.00004 0.00005 0.00007 0.00006 0.00012\n",
      "  0.00008 0.00006 0.00004 0.00006 0.00007 0.00007 0.00017 0.00003 0.00007\n",
      "  0.00004 0.00006]\n",
      " [0.00003 0.00003 0.00004 0.00004 0.00005 0.00005 0.00004 0.00004 0.00004\n",
      "  0.00004 0.00005 0.00004 0.00004 0.00004 0.00004 0.00004 0.00005 0.00005\n",
      "  0.00004 0.00004 0.00004 0.00004 0.00004 0.00004 0.00003 0.00012 0.00005\n",
      "  0.00004 0.00004]\n",
      " [0.00007 0.00007 0.00007 0.00007 0.00007 0.00006 0.00006 0.00008 0.00007\n",
      "  0.00005 0.00007 0.00006 0.00007 0.00004 0.00004 0.00006 0.00006 0.00008\n",
      "  0.00007 0.00006 0.00004 0.00006 0.00008 0.00006 0.00007 0.00005 0.00026\n",
      "  0.00006 0.00005]\n",
      " [0.00004 0.00004 0.00005 0.00004 0.00006 0.00004 0.00004 0.00004 0.00006\n",
      "  0.00004 0.00005 0.00004 0.00004 0.00003 0.00004 0.00004 0.00005 0.00005\n",
      "  0.00005 0.00004 0.00004 0.00004 0.00005 0.00004 0.00004 0.00004 0.00006\n",
      "  0.00015 0.00004]\n",
      " [0.00006 0.00005 0.00007 0.00011 0.00007 0.00013 0.00006 0.00008 0.00005\n",
      "  0.00006 0.00007 0.00005 0.00008 0.00003 0.00004 0.00006 0.00006 0.00007\n",
      "  0.00005 0.00005 0.00004 0.00005 0.00005 0.00006 0.00006 0.00004 0.00005\n",
      "  0.00004 0.00014]]\n"
     ]
    }
   ],
   "source": [
    "#compute mean returns and variance covariance matrix of returns\n",
    "meanReturns = np.mean(arReturns, axis = 0)\n",
    "print('Mean returns:\\n', meanReturns)\n",
    "covReturns = np.cov(arReturns, rowvar=False)\n",
    "#set precision for printing results\n",
    "np.set_printoptions(precision=5, suppress = True)\n",
    "print('Size of Variance-Covariance matrix of returns:\\n', covReturns.shape)\n",
    "print('Variance-Covariance matrix of returns:\\n', covReturns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 4:** ***Prepare parameters for k-means clustering*** \n",
    "\n",
    "Every asset $A_i$ is characterized by its mean return and the variance-covariance vector of its returns with those of  other assets $A_j$. For *i = j*, it would indicate its own variance of returns. Thus the characteristic vector for asset $A_i$ is given by $\\left[\\mu_i, \\sigma_{i1},\\sigma_{i2},...\\sigma_{ii}, ...\\sigma_{iN}) \\right]$, where $\\mu_i$ indicates the mean return of asset $A_i$ and  $\\sigma_{i1},\\sigma_{i2},...\\sigma_{ii}, ...\\sigma_{iN}$ are the variance and covariance of its returns with other assets. It can be seen that $\\left[ \\sigma_{i1},\\sigma_{i2},...\\sigma_{ii}, ...\\sigma_{iN} \\right]$ is nothing but row $i$ of the variance-covariance matrix $V$ of $N$ assets in the stock universe. $\\sigma_{ii}$ which is the variance of the asset return,  is the diagonal element of matrix $V$ in row *i*.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following Python code shows the gathering of parameters for each of the 29 assets in the stock universe.  The parameters are to be provided as inputs to the $k$-means clustering method. Each characteristic vector of the asset comprises 30 components viz.,  its own mean return as the first element of the vector followed by its covariance/variance of returns with the rest of the 29 assets. Thus **assetParameter** holds the the characteristic vectors of all the 29 assets in the stock universe.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of the asset parameters for clustering:\n",
      " (29, 30)\n",
      "Asset parameters for clustering:\n",
      " [[ 0.0009   0.00024  0.00007  0.0001   0.0001   0.0001   0.00007  0.00007\n",
      "   0.0001   0.00007  0.00007  0.00011  0.00005  0.00008  0.00003  0.00005\n",
      "   0.00007  0.00005  0.00012  0.00008  0.00005  0.00004  0.00005  0.00008\n",
      "   0.00007  0.0001   0.00003  0.00007  0.00004  0.00006]\n",
      " [ 0.00029  0.00007  0.00016  0.00008  0.0001   0.00007  0.00006  0.00006\n",
      "   0.00011  0.00007  0.00007  0.00007  0.00005  0.0001   0.00003  0.00004\n",
      "   0.00007  0.00006  0.00008  0.00007  0.00005  0.00003  0.00005  0.00007\n",
      "   0.00007  0.00008  0.00003  0.00007  0.00004  0.00005]\n",
      " [ 0.001    0.0001   0.00008  0.00023  0.00013  0.00009  0.00008  0.00007\n",
      "   0.00011  0.00007  0.00008  0.0001   0.00006  0.0001   0.00004  0.00005\n",
      "   0.00009  0.00006  0.00009  0.00008  0.00005  0.00004  0.00006  0.00007\n",
      "   0.00009  0.00009  0.00004  0.00007  0.00005  0.00007]\n",
      " [ 0.00039  0.0001   0.0001   0.00013  0.00027  0.0001   0.00012  0.00007\n",
      "   0.00013  0.00008  0.00009  0.00012  0.00005  0.00012  0.00004  0.00005\n",
      "   0.00011  0.00006  0.00011  0.00008  0.00006  0.00004  0.00007  0.00007\n",
      "   0.0001   0.0001   0.00004  0.00007  0.00004  0.00011]\n",
      " [ 0.00081  0.0001   0.00007  0.00009  0.0001   0.00018  0.00008  0.00007\n",
      "   0.00009  0.00007  0.00008  0.00012  0.00006  0.00009  0.00004  0.00005\n",
      "   0.00008  0.00006  0.00011  0.00008  0.00006  0.00004  0.00006  0.00007\n",
      "   0.00007  0.00009  0.00005  0.00007  0.00006  0.00007]\n",
      " [ 0.00016  0.00007  0.00006  0.00008  0.00012  0.00008  0.00019  0.00006\n",
      "   0.00009  0.00006  0.00007  0.00008  0.00005  0.00009  0.00004  0.00004\n",
      "   0.00007  0.00006  0.00008  0.00005  0.00005  0.00004  0.00006  0.00006\n",
      "   0.00006  0.00007  0.00005  0.00006  0.00004  0.00013]\n",
      " [ 0.0004   0.00007  0.00006  0.00007  0.00007  0.00007  0.00006  0.00014\n",
      "   0.00008  0.00006  0.00006  0.00007  0.00004  0.00007  0.00003  0.00004\n",
      "   0.00006  0.00005  0.00007  0.00007  0.00005  0.00004  0.00005  0.00006\n",
      "   0.00005  0.00007  0.00004  0.00006  0.00004  0.00006]\n",
      " [ 0.00033  0.0001   0.00011  0.00011  0.00013  0.00009  0.00009  0.00008\n",
      "   0.00021  0.00008  0.00008  0.0001   0.00005  0.00016  0.00003  0.00005\n",
      "   0.00008  0.00007  0.0001   0.00008  0.00006  0.00004  0.00008  0.00008\n",
      "   0.00008  0.0001   0.00004  0.00008  0.00004  0.00008]\n",
      " [ 0.00085  0.00007  0.00007  0.00007  0.00008  0.00007  0.00006  0.00006\n",
      "   0.00008  0.00014  0.00006  0.00007  0.00005  0.00008  0.00003  0.00005\n",
      "   0.00006  0.00005  0.00008  0.00008  0.00005  0.00004  0.00005  0.00007\n",
      "   0.00006  0.00008  0.00004  0.00007  0.00006  0.00005]\n",
      " [-0.00016  0.00007  0.00007  0.00008  0.00009  0.00008  0.00007  0.00006\n",
      "   0.00008  0.00006  0.00016  0.00008  0.00005  0.00008  0.00004  0.00004\n",
      "   0.00007  0.00006  0.00009  0.00006  0.00005  0.00004  0.00005  0.00006\n",
      "   0.00007  0.00008  0.00004  0.00005  0.00004  0.00006]\n",
      " [ 0.00073  0.00011  0.00007  0.0001   0.00012  0.00012  0.00008  0.00007\n",
      "   0.0001   0.00007  0.00008  0.00025  0.00006  0.00009  0.00004  0.00004\n",
      "   0.00009  0.00006  0.00013  0.00007  0.00006  0.00004  0.00006  0.00007\n",
      "   0.00008  0.00009  0.00005  0.00007  0.00005  0.00007]\n",
      " [ 0.00032  0.00005  0.00005  0.00006  0.00005  0.00006  0.00005  0.00004\n",
      "   0.00005  0.00005  0.00005  0.00006  0.0001   0.00005  0.00004  0.00004\n",
      "   0.00006  0.00006  0.00006  0.00005  0.00006  0.00004  0.00005  0.00006\n",
      "   0.00005  0.00005  0.00004  0.00006  0.00004  0.00005]\n",
      " [ 0.0006   0.00008  0.0001   0.0001   0.00012  0.00009  0.00009  0.00007\n",
      "   0.00016  0.00008  0.00008  0.00009  0.00005  0.00017  0.00003  0.00005\n",
      "   0.00008  0.00007  0.00009  0.00007  0.00006  0.00004  0.00008  0.00008\n",
      "   0.00008  0.00009  0.00004  0.00007  0.00004  0.00008]\n",
      " [ 0.00019  0.00003  0.00003  0.00004  0.00004  0.00004  0.00004  0.00003\n",
      "   0.00003  0.00003  0.00004  0.00004  0.00004  0.00003  0.00008  0.00004\n",
      "   0.00004  0.00004  0.00004  0.00004  0.00003  0.00005  0.00004  0.00003\n",
      "   0.00003  0.00004  0.00004  0.00004  0.00003  0.00003]\n",
      " [ 0.00057  0.00005  0.00004  0.00005  0.00005  0.00005  0.00004  0.00004\n",
      "   0.00005  0.00005  0.00004  0.00004  0.00004  0.00005  0.00004  0.00011\n",
      "   0.00004  0.00004  0.00006  0.00005  0.00003  0.00004  0.00004  0.00004\n",
      "   0.00004  0.00005  0.00004  0.00004  0.00004  0.00004]\n",
      " [ 0.00044  0.00007  0.00007  0.00009  0.00011  0.00008  0.00007  0.00006\n",
      "   0.00008  0.00006  0.00007  0.00009  0.00006  0.00008  0.00004  0.00004\n",
      "   0.00012  0.00006  0.00008  0.00006  0.00006  0.00004  0.00006  0.00006\n",
      "   0.00007  0.00007  0.00004  0.00006  0.00004  0.00006]\n",
      " [ 0.00036  0.00005  0.00006  0.00006  0.00006  0.00006  0.00006  0.00005\n",
      "   0.00007  0.00005  0.00006  0.00006  0.00006  0.00007  0.00004  0.00004\n",
      "   0.00006  0.00015  0.00006  0.00005  0.00008  0.00004  0.00005  0.00006\n",
      "   0.00005  0.00006  0.00005  0.00006  0.00005  0.00006]\n",
      " [ 0.001    0.00012  0.00008  0.00009  0.00011  0.00011  0.00008  0.00007\n",
      "   0.0001   0.00008  0.00009  0.00013  0.00006  0.00009  0.00004  0.00006\n",
      "   0.00008  0.00006  0.00021  0.00008  0.00007  0.00005  0.00006  0.00008\n",
      "   0.00008  0.00012  0.00005  0.00008  0.00005  0.00007]\n",
      " [ 0.0008   0.00008  0.00007  0.00008  0.00008  0.00008  0.00005  0.00007\n",
      "   0.00008  0.00008  0.00006  0.00007  0.00005  0.00007  0.00004  0.00005\n",
      "   0.00006  0.00005  0.00008  0.00022  0.00005  0.00004  0.00005  0.00007\n",
      "   0.00006  0.00008  0.00004  0.00007  0.00005  0.00005]\n",
      " [ 0.00034  0.00005  0.00005  0.00005  0.00006  0.00006  0.00005  0.00005\n",
      "   0.00006  0.00005  0.00005  0.00006  0.00006  0.00006  0.00003  0.00003\n",
      "   0.00006  0.00008  0.00007  0.00005  0.00012  0.00004  0.00005  0.00007\n",
      "   0.00005  0.00006  0.00004  0.00006  0.00004  0.00005]\n",
      " [ 0.00025  0.00004  0.00003  0.00004  0.00004  0.00004  0.00004  0.00004\n",
      "   0.00004  0.00004  0.00004  0.00004  0.00004  0.00004  0.00005  0.00004\n",
      "   0.00004  0.00004  0.00005  0.00004  0.00004  0.00009  0.00004  0.00004\n",
      "   0.00004  0.00004  0.00004  0.00004  0.00004  0.00004]\n",
      " [ 0.00043  0.00005  0.00005  0.00006  0.00007  0.00006  0.00006  0.00005\n",
      "   0.00008  0.00005  0.00005  0.00006  0.00005  0.00008  0.00004  0.00004\n",
      "   0.00006  0.00005  0.00006  0.00005  0.00005  0.00004  0.00011  0.00006\n",
      "   0.00006  0.00006  0.00004  0.00006  0.00004  0.00005]\n",
      " [ 0.00095  0.00008  0.00007  0.00007  0.00007  0.00007  0.00006  0.00006\n",
      "   0.00008  0.00007  0.00006  0.00007  0.00006  0.00008  0.00003  0.00004\n",
      "   0.00006  0.00006  0.00008  0.00007  0.00007  0.00004  0.00006  0.00017\n",
      "   0.00006  0.00007  0.00004  0.00008  0.00005  0.00005]\n",
      " [ 0.00019  0.00007  0.00007  0.00009  0.0001   0.00007  0.00006  0.00005\n",
      "   0.00008  0.00006  0.00007  0.00008  0.00005  0.00008  0.00003  0.00004\n",
      "   0.00007  0.00005  0.00008  0.00006  0.00005  0.00004  0.00006  0.00006\n",
      "   0.00013  0.00007  0.00004  0.00006  0.00004  0.00006]\n",
      " [ 0.00101  0.0001   0.00008  0.00009  0.0001   0.00009  0.00007  0.00007\n",
      "   0.0001   0.00008  0.00008  0.00009  0.00005  0.00009  0.00004  0.00005\n",
      "   0.00007  0.00006  0.00012  0.00008  0.00006  0.00004  0.00006  0.00007\n",
      "   0.00007  0.00017  0.00003  0.00007  0.00004  0.00006]\n",
      " [ 0.00023  0.00003  0.00003  0.00004  0.00004  0.00005  0.00005  0.00004\n",
      "   0.00004  0.00004  0.00004  0.00005  0.00004  0.00004  0.00004  0.00004\n",
      "   0.00004  0.00005  0.00005  0.00004  0.00004  0.00004  0.00004  0.00004\n",
      "   0.00004  0.00003  0.00012  0.00005  0.00004  0.00004]\n",
      " [-0.00002  0.00007  0.00007  0.00007  0.00007  0.00007  0.00006  0.00006\n",
      "   0.00008  0.00007  0.00005  0.00007  0.00006  0.00007  0.00004  0.00004\n",
      "   0.00006  0.00006  0.00008  0.00007  0.00006  0.00004  0.00006  0.00008\n",
      "   0.00006  0.00007  0.00005  0.00026  0.00006  0.00005]\n",
      " [ 0.00029  0.00004  0.00004  0.00005  0.00004  0.00006  0.00004  0.00004\n",
      "   0.00004  0.00006  0.00004  0.00005  0.00004  0.00004  0.00003  0.00004\n",
      "   0.00004  0.00005  0.00005  0.00005  0.00004  0.00004  0.00004  0.00005\n",
      "   0.00004  0.00004  0.00004  0.00006  0.00015  0.00004]\n",
      " [-0.00006  0.00006  0.00005  0.00007  0.00011  0.00007  0.00013  0.00006\n",
      "   0.00008  0.00005  0.00006  0.00007  0.00005  0.00008  0.00003  0.00004\n",
      "   0.00006  0.00006  0.00007  0.00005  0.00005  0.00004  0.00005  0.00005\n",
      "   0.00006  0.00006  0.00004  0.00005  0.00004  0.00014]]\n"
     ]
    }
   ],
   "source": [
    "#prepare asset parameters for k-means clustering\n",
    "#reshape for concatenation\n",
    "meanReturns = meanReturns.reshape(len(meanReturns),1)\n",
    "assetParameters = np.concatenate([meanReturns, covReturns], axis = 1)\n",
    "print('Size of the asset parameters for clustering:\\n', assetParameters.shape)\n",
    "print('Asset parameters for clustering:\\n', assetParameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 5:** ***Group the assets into clusters using k-means clustering where k =15, which is the portfolio size selected by the investor.***  \n",
    "  \n",
    "The Python code shows the invocation of the function **KMeans** from the scikit-learn library. \n",
    "The centroids (special points in multidimensional space) towards which the the other physical points representing the asset parameters gravitated to, based on their similarity measure and hence formed a cluster with the centroid as its nucleus,  has been listed in the output. Observe that 15 centroids are obtained for $k=15$. Each point in the multi-dimensional space including the centroids, are of dimension 30.  The labels indicate the cluster to which point $i$ among $N$ points or  asset $i$ of the $N$-stock universe  in reality, belong to. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clustering of assets completed!\n",
      "Centroids:\n",
      " [[ 0.00029  0.00004  0.00004  0.00005  0.00004  0.00006  0.00004  0.00004\n",
      "   0.00004  0.00006  0.00004  0.00005  0.00004  0.00004  0.00003  0.00004\n",
      "   0.00004  0.00005  0.00005  0.00005  0.00004  0.00004  0.00004  0.00005\n",
      "   0.00004  0.00004  0.00004  0.00006  0.00015  0.00004]\n",
      " [ 0.00099  0.0001   0.00008  0.00012  0.0001   0.00009  0.00007  0.00007\n",
      "   0.0001   0.00007  0.00007  0.0001   0.00006  0.00009  0.00004  0.00005\n",
      "   0.00008  0.00006  0.00013  0.00008  0.00006  0.00004  0.00006  0.0001\n",
      "   0.00008  0.00011  0.00004  0.00008  0.00005  0.00006]\n",
      " [-0.00011  0.00007  0.00006  0.00008  0.0001   0.00008  0.0001   0.00006\n",
      "   0.00008  0.00006  0.00011  0.00008  0.00005  0.00008  0.00004  0.00004\n",
      "   0.00007  0.00006  0.00008  0.00005  0.00005  0.00004  0.00005  0.00005\n",
      "   0.00006  0.00007  0.00004  0.00005  0.00004  0.0001 ]\n",
      " [ 0.00034  0.00005  0.00005  0.00006  0.00006  0.00006  0.00005  0.00004\n",
      "   0.00006  0.00005  0.00005  0.00006  0.00007  0.00006  0.00004  0.00004\n",
      "   0.00006  0.0001   0.00006  0.00005  0.00009  0.00004  0.00005  0.00006\n",
      "   0.00005  0.00006  0.00004  0.00006  0.00004  0.00005]\n",
      " [ 0.00082  0.00008  0.00007  0.00008  0.00008  0.00008  0.00006  0.00006\n",
      "   0.00008  0.00011  0.00006  0.00007  0.00005  0.00007  0.00004  0.00005\n",
      "   0.00006  0.00005  0.00008  0.00015  0.00005  0.00004  0.00005  0.00007\n",
      "   0.00006  0.00008  0.00004  0.00007  0.00005  0.00005]\n",
      " [ 0.00039  0.0001   0.0001   0.00013  0.00027  0.0001   0.00012  0.00007\n",
      "   0.00013  0.00008  0.00009  0.00012  0.00005  0.00012  0.00004  0.00005\n",
      "   0.00011  0.00006  0.00011  0.00008  0.00006  0.00004  0.00007  0.00007\n",
      "   0.0001   0.0001   0.00004  0.00007  0.00004  0.00011]\n",
      " [ 0.00077  0.0001   0.00007  0.00009  0.00011  0.00015  0.00008  0.00007\n",
      "   0.0001   0.00007  0.00008  0.00018  0.00006  0.00009  0.00004  0.00005\n",
      "   0.00008  0.00006  0.00012  0.00008  0.00006  0.00004  0.00006  0.00007\n",
      "   0.00008  0.00009  0.00005  0.00007  0.00005  0.00007]\n",
      " [ 0.00017  0.00007  0.00007  0.00009  0.00011  0.00008  0.00013  0.00006\n",
      "   0.00009  0.00006  0.00007  0.00008  0.00005  0.00008  0.00004  0.00004\n",
      "   0.00007  0.00005  0.00008  0.00006  0.00005  0.00004  0.00006  0.00006\n",
      "   0.0001   0.00007  0.00004  0.00006  0.00004  0.00009]\n",
      " [ 0.00042  0.00006  0.00006  0.00007  0.00008  0.00007  0.00006  0.00008\n",
      "   0.00008  0.00006  0.00006  0.00007  0.00005  0.00008  0.00004  0.00004\n",
      "   0.00008  0.00005  0.00007  0.00006  0.00005  0.00004  0.00007  0.00006\n",
      "   0.00006  0.00007  0.00004  0.00006  0.00004  0.00006]\n",
      " [ 0.00022  0.00003  0.00003  0.00004  0.00004  0.00004  0.00004  0.00004\n",
      "   0.00004  0.00004  0.00004  0.00004  0.00004  0.00004  0.00006  0.00004\n",
      "   0.00004  0.00004  0.00005  0.00004  0.00004  0.00006  0.00004  0.00003\n",
      "   0.00004  0.00004  0.00007  0.00004  0.00004  0.00004]\n",
      " [ 0.00057  0.00005  0.00004  0.00005  0.00005  0.00005  0.00004  0.00004\n",
      "   0.00005  0.00005  0.00004  0.00004  0.00004  0.00005  0.00004  0.00011\n",
      "   0.00004  0.00004  0.00006  0.00005  0.00003  0.00004  0.00004  0.00004\n",
      "   0.00004  0.00005  0.00004  0.00004  0.00004  0.00004]\n",
      " [-0.00002  0.00007  0.00007  0.00007  0.00007  0.00007  0.00006  0.00006\n",
      "   0.00008  0.00007  0.00005  0.00007  0.00006  0.00007  0.00004  0.00004\n",
      "   0.00006  0.00006  0.00008  0.00007  0.00006  0.00004  0.00006  0.00008\n",
      "   0.00006  0.00007  0.00005  0.00026  0.00006  0.00005]\n",
      " [ 0.00031  0.00008  0.00014  0.00009  0.00011  0.00008  0.00008  0.00007\n",
      "   0.00016  0.00007  0.00007  0.00009  0.00005  0.00013  0.00003  0.00004\n",
      "   0.00007  0.00006  0.00009  0.00007  0.00006  0.00004  0.00007  0.00008\n",
      "   0.00008  0.00009  0.00004  0.00007  0.00004  0.00007]\n",
      " [ 0.0006   0.00008  0.0001   0.0001   0.00012  0.00009  0.00009  0.00007\n",
      "   0.00016  0.00008  0.00008  0.00009  0.00005  0.00017  0.00003  0.00005\n",
      "   0.00008  0.00007  0.00009  0.00007  0.00006  0.00004  0.00008  0.00008\n",
      "   0.00008  0.00009  0.00004  0.00007  0.00004  0.00008]\n",
      " [ 0.0009   0.00024  0.00007  0.0001   0.0001   0.0001   0.00007  0.00007\n",
      "   0.0001   0.00007  0.00007  0.00011  0.00005  0.00008  0.00003  0.00005\n",
      "   0.00007  0.00005  0.00012  0.00008  0.00005  0.00004  0.00005  0.00008\n",
      "   0.00007  0.0001   0.00003  0.00007  0.00004  0.00006]]\n",
      "Labels:\n",
      " [14 12  1  5  6  7  8 12  4  2  6  3 13  9 10  8  3  1  4  3  9  8  1  7\n",
      "  1  9 11  0  2]\n"
     ]
    }
   ],
   "source": [
    "#kmeans clustering of assets using the characteristic vector of \n",
    "#mean return and variance-covariance vector of returns\n",
    "\n",
    "assetsCluster= KMeans(algorithm='auto',  max_iter=600, n_clusters=clusters)\n",
    "print('Clustering of assets completed!') \n",
    "assetsCluster.fit(assetParameters)\n",
    "centroids = assetsCluster.cluster_centers_\n",
    "labels = assetsCluster.labels_\n",
    "\n",
    "print('Centroids:\\n', centroids)\n",
    "print('Labels:\\n', labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 6:**  ***Fix asset labels to points in each cluster***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stocks in each of the clusters:\n",
      "\n",
      "Cluster 1\n",
      "['WMT']\n",
      "Cluster 2\n",
      "['BA' 'MSFT' 'UNH' 'V']\n",
      "Cluster 3\n",
      "['IBM' 'XOM']\n",
      "Cluster 4\n",
      "['JNJ' 'MRK' 'PFE']\n",
      "Cluster 5\n",
      "['HD' 'NKE']\n",
      "Cluster 6\n",
      "['CAT']\n",
      "Cluster 7\n",
      "['CSCO' 'INTC']\n",
      "Cluster 8\n",
      "['CVX' 'UTX']\n",
      "Cluster 9\n",
      "['DIS' 'MMM' 'TRV']\n",
      "Cluster 10\n",
      "['KO' 'PG' 'VZ']\n",
      "Cluster 11\n",
      "['MCD']\n",
      "Cluster 12\n",
      "['WBA']\n",
      "Cluster 13\n",
      "['AXP' 'GS']\n",
      "Cluster 14\n",
      "['JPM']\n",
      "Cluster 15\n",
      "['AAPL']\n"
     ]
    }
   ],
   "source": [
    "#fixing asset labels to cluster points\n",
    "print('Stocks in each of the clusters:\\n',)\n",
    "assets = np.array(assetLabels)\n",
    "for i in range(clusters):\n",
    "    print('Cluster', i+1)\n",
    "    clt  = np.where(labels == i)\n",
    "    assetsCluster = assets[clt]\n",
    "    print(assetsCluster)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be seen that the 29 assets of the stock universe have been grouped into 15 clusters. The idea conveyed is that all assets in the same cluster behave **similar**   (*inter-cluster similarity*) with regard to their mean and variance-covariance of returns, and are **dissimilar** (*intra-cluster dissimilarity*) with regard to the same characteristics with those assets in other clusters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore picking **one asset from each cluster** to gather a portfolio of 15 assets would ensure that the portfolio is well-diversified with regard to these characteristics. The choice could be random or preferential, but restricted to one asset from each cluster. For ease of reference,  we term the choices made as **$k$-portfolio**. Needless to say, multiple $k$-portfolios can be generated from these clusters. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since $k$-means clustering is a heuristic method which produces clusters that are sensitive to the randomly chosen initial centroids, it may yield different cluster configurations with each run. In practice, an aggregation of the clusters yielded during various runs can be studied to make the appropriate choice of assets one each from each cluster to construct the well-diversified portfolio."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5 $k$-portfolios - a brief note"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us suppose that a specific run of the $k$-means clustering algorithm yielded the following clusters for the DJIA index \"mini\" universe:\n",
    "[KO PG  VZ  WMT], [UNH], [DIS  MMM  TRV], [IBM, XOM], [CSCO  INTC], [JPM], [GS], [WBA], [AAPL  MSFT  V], [HD  NKE], [AXP  CVX  UTX], [MCD],[JNJ  MRK  PFE], [BA], [CAT]  \n",
    "  \n",
    "The investor is now free to make a choice of one asset each from each of the clusters. The choice could be random or guided by individual preferences. The following  $k$-portfolios are some sample choices that can be made by investors.  \n",
    "\n",
    "**$k$-portfolio 1**:  \n",
    "{ [KO], [UNH], [DIS], [IBM], [CSCO], [JPM], [GS], [WBA], [AAPL], [HD], [AXP], [MCD], [MRK], [BA], [CAT] }  \n",
    "\n",
    "**$k$-portfolio 2**:  \n",
    "{ [VZ], [UNH], [MMM], [XOM], [INTC], [JPM], [GS], [WBA], [MSFT], [NKE], [CVX], [MCD], [PFE], [BA], [CAT]}  \n",
    "\n",
    "**$k$-portfolio 3**:  \n",
    "{ [WMT], [UNH], [TRV], [IBM], [CSCO], [JPM], [GS], [WBA], [V], [NKE], [UTX], [MCD], [JNJ], [BA], [CAT]}  \n",
    "\n",
    "An investor can  opt to invest in any one of the $k$-portfolios. Fig. 3.5 illustrates heuristic selection of portfolios where the portfolio of assets is diversified in behaviour adhering to the adage \" never put all eggs in the basket\". "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](Lesson3Fig3_5.png)\n",
    "####  Fig. 3.5 Heuristic selection of assets to construct a diversified portfolio "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, several questions do arise on the behavior of the $k$-portfolios, their risk-return tradeoffs and their performance when time tested portfolio construction techniques are applied over them. These shall be covered in detail in the ensuing lessons. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Companion Reading \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This work  is an abridged adaptation of concepts discussed in Chapter 3 of [PAI 18] to Dow Jones dataset (DJIA Index: April, 2014-April, 2019) and implemented in Python using NumPy, Pandas and scikit-learn libraries. Readers (read \"worker bees\") seeking more information may refer to the corresponding chapter in the  book."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align=\"left\">References</h3>  \n",
    "\n",
    "[CHO 08]   Choueifaty Yves and  Y Coignard, Toward Maximum Diversification, *The Journal of  Portfolio Management*, pp. 40-51, 2008.\n",
    "\n",
    "[CHO 13]   Choueifaty  Yves, T Froidure and J Reynier, Properties of the Most Diversified Portfolio, *Journal of Investment Strategies*, 2(2), pp. 49-70, 2013.  \n",
    "  \n",
    "[PAI 18]   Vijayalakshmi Pai G. A., Metaheuristics for Portfolio Optimization- *An Introduction using MATLAB*, Wiley-ISTE, 2018. https://www.mathworks.com/academia/books/metaheuristics-for-portfolio-optimization-pai.html \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Next.....Lesson 4: Traditional Methods for Portfolio Construction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](Lesson3ExitTailImage.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
